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ABSTRACT 

Fourteen novel multiplication problems (of the form 
^*a times b equals") were presented on each of five consecutive days 
to nine third grade classes divided into three groups. Immediate 
knowledge of results was provided for group one, delayed for group 
two, and no knowledge of results for group three. Residual scores, 
obtained by taking the difference between the number of novel 
problems answered correctly and the predicted number usi!ig stepwise 
multiple regression, were used as the dependent variables. These 
residual scores were analyzed using a four factor luixed model 
analyses of variance, with subjects nested within classes which in 
turn were nested within treatments. The fourth factor was a repeated 
measures factor (acquisition and retention tests for atialyses one, 
treatment days for analysis two, and performance on non-novel 
problems for analysis three). No significant treatment effects were 
found* Significant differences among classes were observed, pointing 
out clearly hov this source of variation, which has historically been 
implicitly pooled by investigators unfamiliar with statistical 
nesting, can affect scores* (Author/FL) 
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The Effect of Delay of Knowledge 
of Results on the Acquisition and 
Retention of Novel Multiplication Facts 
VJ. Todd Rogers 
Kenneth D, Hopkins 
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University of Colorado 



Since the early fifties a number of experimental studies concerned 
with the effect of delay of knowledge of results on learning in human 
beings have appeared in the psychological literature. Prior to this 
time, delay of reinforcement had largely been confined to the study 
of lower animals, principally rats. The results of these rat studies 
suggested that delay of reinforcement irapalrsd th3 acquisition of 
new learning, with no predictable outcome for the retention of the 
material learned (Renner, 196A). There is no such consensus in 
studies of human learning. 

Several studies reported between 1955 and 1960 failed to find any 
deleterious effects of delayed knowledge of results on the speeds at 
which college students learned various siirple motor tasks (Saltemani 
Kanfcr and Greenspoon, 1955; Bilodeau and Bilodeau^ 1956; Kc Guigani 
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1959; Bilodeau and Ryan, I960; Denny, Allard, Hall and Rokeach, 

1960) « The typical range of delay used in these studies wsf from 
0 to 30 seconds. Bilodeau and Bilodeau (1958) and Denny (1960) 

have concluded, as a result of their findings, that the interaction 
between the intertrial interval sad the post-knowledge of results 
Interval was a more Important variable than the delay interval. 

In the experiments where a concept identification task or memory 
was involved, the results are contradictory. Saltzman (1951) 
Investigated the effect of delaying knawledge of results on the rate 
of learning a list of fcur place numerals presented in a verbal 
maze. Using college students, he found that immediate reinforcement 
led to a higher rate of learning than a delay of six seconds. Sax 
(1960) found that an increased delay in knowledge of results (no 
delay, 10 minutes, 20 minutes, and ^*0 minutes) led to a significant 
increase in the number of trials needed by high school students to 
learn a series of Chinese symbols by the paired-associate method. 
Similarly, Bourne (1957) reported that the performance or students 
of an elementary psychology class, presented with the task of 
Identifying a series of geometric patterns, was inversely related to 
the delay of feedback (0 to 8 seconds). However, in a later study 
(Bourne and Bunderson, 1963) t in which the post-delay interval was 
kept constant, findings ainilar to those reported in the simple 
motor learning experiments were obtained. 

Two studies by Drackblll and her collaborators (Brackbill and 
Kappy, 1962a; Brackbill, Isaacs and Smelkinson, 1962b) failed to 
show that acquisition of a jet of two choice discriminations by 
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third grade boys was impaired by delay of reinforcement (0^ and 
10 seccnds). Markowitz and Renner (1966) proposed that the effect 
reported by Brackbill was due to feedback which vas given iti addition 
to the reinforcement. To verify their proposal, they replicated the 
discrimination procedures used by Brackbill, providing different 
combinations of feedback and immediate or delayed reinforcement. They 
found that when feedback is eliminated and reinforcement delayed, 
the number of trials to criterion increased significantly. 

The studies of Sox, Brackbill, Isaacs, and Stnelkinson, and Markowitz 
and Renner investigated the effect of delaying knowledge of results on 
retention* \s for acquisition, the results obtained were contradictory. 
Sax (1960) concluded that deley of knowledge of results (0 seconds, 
lO minutes, 20 minutes, and 40 minutes) during acquisition was 
ineffective when retention (two week retention interval) was employed 
as the dependent variable. Brackbill and others (1962a, 1962b) found 
that delaying knowledge of results by 10 seconds during acquisition 
facilitated retention over the short interval (one day), but that this 
effect faded over the longer interval (eight days), Markowitz and 
Kenner ^&966) reported that retention by subjects given delayed 
feedback was significantly greater than by subjects having no feedback. 
For subjects having reinforcement, immediate or delayed, differences 
in retention were not significant. Markowitz and Renner concluded that 
the delayed retention effect reported by Brackbill was a function of 
the feedback present in her studies. 

It is apparent that present knowledge regarding the effect of 
delay of knowledge of results on humar xt rning is inconclusive. 
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Most studies have Involved experimental controls» with an ^perimenter 
working individually with each subject. As a result » the external 
validity of the research findings to the classroom Is uncertain. The 
aim of this study is to determine the effect of delay of knowledge 
of results on the learning of novel mathematical facts through drill 
in a classroom situation. Specifically, are there significant 
differences In the acquisition and retenti:>n of novel multiplication 
facts between groups of third-grade students provided with immediate 
Knowledge of results, delayed knowledge of results, and no knowledge 
of results? 



Method 

Subjects 

Nine classes of heterogeneously grouped third-grade pupils from 
five elementary schools were used in this study. The schools, located 
in a large metropolitan area, ware selected on the basis of their 
sire (each school except one contained two third-grade classes) and 
their proximity to each other. 

Two factors were involved in choosing third-grade students as 
the population from which the experimental groups could b'. drawn. 

First 9 pupils of this grade were comparable to subjects used in 
previous psychological studies* notably those conducted by Brackbill 
and her collaborators* imd by Markowitz and Renner. Second* suitable 
educational roatorial which the students had not been exposed to and yet 
for which the basic groundwork had been laid could be found. In particular* 
multiplication of the natural numbers begins in the third grade* with 
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those multiplication facts having products of 36 or less being 
developed as the year progresses. 

Altogether 219 students o£ mean age 107.7 months and mean l.Q. 
of 107.2 (California Short-Form Test of Mental Maturity t Form IH) 
were present for the entire experiment. None of these students 
possessed a hearing or speech disability. 

The classes were assigned to the three treatments using a 
stratified ordering procedure. To partially balance school variance « 
no treatment was replicated in the same school. In addition, the 
treatments were distributed, so far as was possible, evenly throughout 
the day. An equal number of pupils per class (n ■* 15) was randomly 
selected from each of the nine classes. The assignment of treatments 
to classes is summarized in Table I. 

Materials 

Multiplication test . The multiplication test involved a reading of 
all the problems of the form ^'a x b", and *'b" natural numbers 
between 1 and 10. The problems were arranged In a random order, 
each separated from the next by a response interval of three seconds. 

The oral presentation of a problem consisted of the number of the 
problem followed by the problem. For example, the test began: 

'^Number 1: .3 tiroes 3 equals (3 sec.). Humber 2; 5 times 7 equals 
(3 sec.).*' Each student was provided with an answer sheet on which 
he could record his answers. Since the test was to be administered 
on Ihree different occasions to each of the nine classes, the po6s^bility 
arose of variations in scores due to changes in delivery (for example, 
vvice inflexion) and tlning. To avoid this source of variance, a tape 
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recording of the test was used. 

Novel CTuX t lpllcfltlo n facts , All of the students of each class 
were pretested using the multiplication test to ascertain their 
knowledge of the multiplication facts. Those problems which had a 
difficulty index of .05 or 1 3S were considered to be the novel 
problems. Fourteen such problems were found. 

The fourteen novel problems were presented twice on each of 
five consecutive days. To avoid the possibility of serial learning, 
the problems were presented In a different random order on each day# 
The fourteen problems together with their difficulty indices (p) and 
daily orders of presentation are presented in Table 2. 

Tlie problems, each separated from the next by the response 
interval, were presented orally by the experimenter. For the classes 
provided with immediate knowledge of results, the presentation 
consisted of the number of each problem, the problem, a response 
interval, and the corresponding multiplication fact. For example, 
'^Number 1: t times 7 equals (3 sec.). 6 times 7 equals 42. Number 
2: 6 times 8 equals (3 sec.). 6 times 8 equals 48, etc," foe 
the classic provided ulth delayed knowledge of results, the oral 
presentation consisted of the number of each problem, followed by the 
problem and response interval* Upon completion of both trials, the 
multiplication facta together with the corresponding number were read. 
For the classes provided with no knowledge of results, the oral 
presentation consisted of the number of each problem, followed by 
the problem and response interval. 

Since each presentation was to be made to three classes, the 
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Novel ^fciltipllcation Facts: Difficulty Indices 
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possibility arose of wlthiu-treatment variations due to changes lu 
delivery and tlxaing. To avoid this source of variance three tapes 
were used each day corresponding to the treatments and orders of 
presentation of the problems. 



Procedure 

On Day 1, the orientation day, the experimenter explained to the 
students of each class that they would be taking part in an experiment 
to determine hew pupils of the third grade learn arithmetic. A trial 
run, using ten addition facts, was conducted to acquaint the scudents 
with t\\c materials and procedures to be followed. 

The pupils were pretested using the multiplication test on the 
second day to determine (a) their knowledge of the multiplication 
facts and (b) the novel m/dtlplication facts. Before beginning the 
test, the students were told It w£s a test of multiplication and 
that some of the questions would be easy, while others would be 
quite difficult . 

Day 3, tho first of five consecutive treatment days, began with 
each student being provided with a study sheet of the fourteen novel 
facts to be learned. The facts were read aloua in the order in which 
they appeared on the study sheet. The students were then instructed 
CO study the tacts for a quia to be given five minutes later. After 
the study ;>eriod, the study sheets were collected and kept b> the 
experimenter. Each class then received the appropriate tape for Day 
3, the students recording their responses on the answer sheets 
provided. Both groups provided with knowledge of results were encouraged 
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to mark their own papers, placing a ''tick'* beside an answer that was 
correct and a "cross" beside an incorrect answer or a space left 
blank. Students who were observed filling In the blanks during the 
reading of the answers vere eliminated from the statistical analyses. 
Upon completion of each tape, the answer sheets were collected and 
kept by the experimenter. 

The treatments were repeated on Days 4 through 7. After the 
tape on the seventh day, the multiplication test was re-adrolnlstered 
to obtain a measure of acquisition. Six calendar days later the 
classes were retested using the multiplication test to obtain a measure 
of retention, During the retention Interval, no instruction In 
multiplication facts, novel or non-novel, was given. The students 
were told, beofre taking the test, che the purpose of the test was 
tc see how much they could remember from the previous week's work. 




Analysis 

Data 

I.Q. scores and ages In months as of June 1 were collected from 
the permanent record cards of each student who participated In the 
experiment. From the three administrations of the multiplication 
test (pretest, acquisition teat, and retention test) six different 
scores, corr^.ipondlng to the number of novel problems answered 
correctly on «!ach administration and to the number of non-novel 
problems answered correctly on each administration, were obtained 
for each studctnt. In addition, five scores vere obtained corresponding 
to the number of problems answered correctly on each of the treatment 
days. On each treatment day, a problem was marked correct only If 
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the student provided the proper answer on both presentations. 
Acquisition and Retention 

Acquisition was defined rs the difference between the number of 
novel probl^.ms answered correctly on the second administration of 
the multiplication test and the number predicted using stepwise 
multiple regression, etrploylng I.Q., age, and total pretest score 
(total number of novel and non-novel problems answered correctly 
on the first administration of the problems) as predictors. To 
assess differential retention, the difference between the score tor 
the third administration of the multiplication test and the predicted 
fCore (using the three predictors indicated above) was used. For 
both acquisition and retention the stepwise regression selected the 
pretest score as the best predictor with the remaining two predictors 
accounting for less than one additional percent of the variance. 
Because little was to be gained from the Inclusion of all three 
predictors, only the pretest score was uaed for the analyses* 

A four factor mixed model analysis of variance was employed in 
which 13 subjects were nested within each of the three classes, which 
were in turn gested wit)hin each of the three treatments. The 
fourth factor, acquisition and retention test scores, waa a 
repeated-measures factor. (The pupil and class fsetors were treated 
as random factors, the other two being fixed.) l1ie true experimental 
unit was classes, the otse^-vationsl unit being pupil test scores, 
following the procedure suggested by Addelman (1970). For the 
repeated-measures factor, tlu procedure suggested by Greenhouse and 
Gelaser 0959) fir detemlnlng the significance of the obtained F 
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was u&ed. A pooling strategy was enployp.d in order to maximize 
power: alpha was set at .10 for the nested factor, hence If the null 
hypothesis for the difference among class means within treatments 
was tenable at this levels this source of variation would be pooled 
with the subjects withir treatments and classes » thereby increasing 
the power for testing the main treatment effect. The results of 
the analysis are given in Table 3. Since this criterion was not achieved» 
pooling was not legitimized or employed. 



Table 3 

Analysis of Variance: Ac<]ui8ition and Retention 





Source of Variation 


Mean 

Square 


F 



Treatment 




2 


21.65 


1.31 


Classes vithin treatment 




6 


16. (il 


1.96* 


Students with treatment 


and classes 


126 


8.39 




Tests^ 




1 


.00 




Treatment X test 




2 


.02 




Vests X classes within 


treatment 


6 


2.24 


1.47 


Tests X students within 
classes 


treatment and 


126 


1.52 





a 

Since residual gain scores served as dependent variablesi 
a mean s<iuare value of .00 will always result. The hypotheses 
involving interactions with testa > however » are meaningful. 

*P < .10 
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As shown in Table there were no significant differences 
among the three treatment groups for acquisition or retention 
(F ■ 1.31) and between acquisition and retention <F<1.0), nor 
any significant interactions. There was a significant difference 
among the classes nested within treatments (F » 1.96, p <. 10)9 
suggesting that class means differed significantly even within the 
treatme : Because the main interest of this study was with acquisition 

and retention of multiplication facts under three different delay 
of knowledge of results conditions j no further attention was directed 
toward the classroom effect and corresponding Interactions involving 
this effect. 

Learning Patterns 

A second four factor mixed model analysis of variance was 
performed to examine the learning patterns across the five treatment 
days. Pupils and classes were nested within treatments in the same 
manner as in the previous analysis. Days were treated as a repeated- 
measures factor. The dependent variable for this analysis was the 
«:ifference between the number of novel problems ar.svered correctly on 
both presentotions of the problem for each day and the number 
predicted using the total pretest score as the predictor. The 
results of the analysis of variance are included in Table A. 
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Table 4 

Analysis of Variance: Daily Learning Patterns 



■ 


Source of Variation 


df 

Square 


F 





Treatment 


2 


496.89 


4.46 


Classes within treatment 


6 


111.31 


5.45* 


Subjects within treatments and classes 


126 


20.42 




Days^ 


4 


.00 




Treatment X days 


8 


3.44 


.90 


Days X classes within treatments 


24 


3.83 


1.47* 


Days X subjects within treatments and 
classes 


504 


2.60 





Since residual gain scores served as dependent variables, 
a mean square value of .00 wllx always result* The hypothesis 
involving interaction with days, however, are meaningful. 

A 

p < .10 

The results from this analysis are similar to those reported 
earlier for acquisition and retention, with the only significant 
resulted involving classes nested within treatments. Treatment 
means did not differ signiti^rantly . 

Non-novel iacts 

The data collected were amenable to investigating possible 
effects produced by the thiee treatments on performance on the 
non-novel (and iience not taught) multiplication facts. The difference 
between the number of non^novel probleins answered correctly on the 
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second administration of the multiplication test and the lumber 
predicted using total pretest score aa the predictor was defined as 
the short term effect; the difference between the number of non* 
novel problems answered correctly on the third administration of the 
test and the number using total pretest score as predictor was 
defined as the long term effect. These effects were analyzed 
using the ^ame analysis procedure as for acquisition and retention. 
The results of this analysis are reported in Table 5. 

Table 5 

Analysis of Variance: Short and Long Term Effects 



Source of Variation 


df 


Mean 

Square 


F 


Treatment 


2 


232. 7S 


1.82 


Classes wi thlti treatmer.t 


6 


128.18 


1.95* 


Subjects vithin classes and treatment 


126 


65.76 




Length of effect (LOE)® 


1 


.00 




Treatment X LOE 


2 


2.36 


.70 


LOE X clashes within treati&ent 


6 


33.78 


2.09* 


LOE X subjects within treatments and 
class 


126 


16.16 





Q 

Since residual gain scorea served as dependent variables , 
a mean square value of .00 will always result The hypothesis involving 
interactions with length of effect, however, are meaningful. 

* P < .10 



o 

ERIC 



15 



16 



As shown in Table 3| rhe only significant results obtained are 
for classes nested within treatments and for the interaction between 
classes within treatment and length of effect* 

Discussion 

The study was designed to investigate the effect of delay of 
knowledge of results on the acquisition and retention of novel 
multiplication facts presented in a classroom situation. The 
findings obtained failed to support the generalization that where 
knowledge of results is given to one groups either immediately or 
delayed 9 and withheld from a second , comparable groupi the former 
will reach a higher level of proficiency. The analysis previously 
reported indicated that the differences in the means among treatment 
groups were not significant. The apparent lack in differences 
among the three treatments was likely due to the extreme difficulty 
of the tabk for the third grade pupils. The mean scores for all 
groups were about 12**13 percent of the 14 novel problems for both the 
acquisition and retention teats. Failure to achieve higher scores 
was attributed to the short treatment period which leaves unanswered 
the question of whether or not any advantage would be gained by 

(1) repeating the treatmente over a longer period of time, and 

(2) increasing the number of presentations of each problem at one 
time. 

The three treatments utilised in the study involved differences 
in intertrial intervals as well as differences in delay intervals. 

It may well be that the results obtained were more a function of 
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the Intertrlal interval than of the delay interval. Further 9 tudy is 
needed to determine the effect of intervening material during the 
response interval » as was the case for the delayed group. Would 
a diffeiential effect be observed if the delayed treatment involved a 
delay of eight seconds after the response interval » with no intervening 
material? In such an experiment, would the results be confounded 
with possible covert practice? 

The significant classroom effect » a nuisance variable which 
has historically been implicitly po:>lcd by investigators unfamiliar 
with statistical nesting, points out clearly how this source of 
variation ran affect scores. Had this factor been ignored in the 
present study, the treatment results would have been erroneously 
Judged to be statistically significant. Ideally, each treatment 
should be administered in each classroom to randomly selected 
subgroups, thus having classroom as a crossed, rather than as a 
nested, factor. 
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Table 6 

Means and Standard Deviationa: IQ^ 





Treatment Class 


X 


S 



I. 


Immediate Knowledge of Results 
1 


114.73 


11.28 




2 


106.73 


15.82 




3 


111.00 


6.72 


II. 


Delayed Knowledge of Keaults 
4 


106.87 


14.32 




5 


109.27 


10.73 




6 


105.93 


12.38 


III. 


No Knowledge of Results 
7 


110.33 


14.44 




8 


104.20 


12.16 




9 


104.67 


16.30 



^California Short-Form Teat of Mental Maturity, Form IH* 




21 



Table 7 

Means and Standard Deviations: Age^ 





Treatment Class 


X 


S 



I. 


Immediate Kncn/ledge of Results 
1 


106.53 


A. 19 




2 


106.93 


6.03 




3 


107.87 


3.38 


II. 


Delayed Knowledge of Results 
4 


108.73 


2.19 




5 


107. AO 


0.48 




6 


107. AO 


5.05 


III. 


No Knowledge of Results 
7 


106.53 


5.40 




8 


110.33 


6.99 




9 


107.53 


2.85 



a 



AKa In months as of June 



1 . 
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Tfblc 8 



Means and Standard Deviations; Pretest 







Treatment Class 


X 


X 



I. Immediate Knowledge of Res Jits 






1 


18.40 


7.18 


2 


29.93 


10.83 


3 


33.13 


11.25 


II* Delayed Knowledge of Results 






A 


33.60 


12.36 


5 


31.00 


7.34 


6 


27.60 


8.51 


III. No Kncneiedge of Results 






7 


31.87 


15.32 


8 


18.27 


10.33 


9 


24.87 


9.57 
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Table 9 



Means and Standard Deviations: Novel Problems on 
Acquisition and Retention Tests 





Treatment Class 


Acquisition 
X S 


Retention 
X S 



I. 


Immediate Knowledge of Results 
1 


1.80 


1.97 


1.87 


2.36 




2 


2.80 


2.96 


3.07 


2.63 




3 


2.00 


2.56 


2.40 


3.00 


IX. 


Delayed Knowledge of Results 
4 


1.33 


1.40 


2.60 


4.51 




5 


2.33 


2.72 


2.27 


3.24 




6 


1.33 


2.69 


1.00 


1.93 


III. 


No Knowledge of Results 
7 


3.00 


3.93 


3.53 


3.96 




6 


.13 


.35 


.20 


.56 




9 


.73 


1.62 


.40 


.74 
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Table 10 



Keans and Standard Deviations; Non*novel Troblems on 
Acquisition and Retention Tests 



Treatment Class Acquisition Retention^ 



1. 


loirediate Knowledge of Results 
I 


20.53 


9.12 


21.67 


8.19 




2 


30.33 


11.57 


33.80 


12.60 




3 


36.07 


10.16 


37.87 


10.23 


II. 


Delayed Knowledge of Refiults 
4 


31.87 


12.14 


31.33 


12.66 




5 


30.87 


7.86 


33.53 


8.73 




6 


29.87 


10.88 


31.07 


12.42 


III. 


No Knowledge of Results 
7 


29.00 


16.68 


27.07 


17.08 




8 


18. AO 


11.98 


20.40 


12.38 




9 


2A.60 


12.34 


28.67 


11.01 
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and Standard Dcrvlatlons; Novel Problem? on Treatment Days 
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A problem was considered correct only if It vas answered correctly on both presentations 
of che problem* 



